Devices for imaging body cavities or passages in vivo are known in the art and include endoscopes and autonomous encapsulated cameras. Endoscopes are flexible or rigid tubes that pass into the body through an orifice or surgical opening, typically into the esophagus via the mouth or into the colon via the rectum. An image is formed at the distal end using a lens and transmitted to the proximal end, outside the body, either by a lens-relay system or by a coherent fiber-optic bundle. A conceptually similar instrument might record an image electronically at the distal end, for example using a CCD or CMOS array, and transfer the image data as an electrical signal to the proximal end through a cable. Endoscopes allow a physician control over the field of view and are well-accepted diagnostic tools.
Capsule endoscope is an alternative in vivo endoscope developed in recent years. For capsule endoscope, a camera is housed in a swallowable capsule, along with a radio transmitter for transmitting data, primarily comprising images recorded by the digital camera, to a base-station receiver or transceiver and data recorder outside the body. The capsule may also include a radio receiver for receiving instructions or other data from a base-station transmitter. Instead of radio-frequency transmission, lower-frequency electromagnetic signals may be used. Power may be supplied inductively from an external inductor to an internal inductor within the capsule or from a battery within the capsule.
An autonomous capsule camera system with on-board data storage was disclosed in the U.S. Pat. No. 7,983,458, entitled “In Vivo Autonomous Camera with On-Board Data Storage or Digital Wireless Transmission in Regulatory Approved Band,” granted on Jul. 19, 2011. The capsule camera with on-board storage archives the captured images in on-board non-volatile memory. The capsule camera is retrieved upon its exiting from the human body. The images stored in the non-volatile memory of the retrieved capsule camera are then accessed through an output port on in the capsule camera.
While the two-dimensional images captured by the endoscopes have been shown useful for diagnosis, it is desirable to be able to capture gastrointestinal (GI) tract images with depth information (i.e., three-dimensional (3D) images) to improve the accuracy of diagnosis or to ease the diagnosis process. In the field of 3D imaging, 3D images may be captured using a regular camera for the texture information in the scene and a separate depth camera (e.g. Time of Flight camera) for the depth information of the scene in the field of view. The 3D images may also be captured using multiple cameras, where multiple cameras are often used in a planar configuration to capture a scene from different view angles. Then, point correspondence is established among multiple views for 3D triangulation. Nevertheless, these multi-camera systems may not be easily adapted to the GI tract environment, where the space is very limited. In the past twenty years, a structured light technology has been developed to derive the depth or shape of objects in the scene using a single camera. In the structured light system, a light source, often a projector is used to project known geometric pattern(s) onto objects in the scene. A regular camera can be used to capture images with and without the projected patterns. The images captured with the structured light can be used to derive the shapes associated with the objects in the scene. The depth or shape information is then used with regular images, which are captured with non-structured floodlit light, to create 3D textured model of the objects. The structured light technology has been well known in the field. For example, in “Structured-light 3D surface imaging: a tutorial” (Geng, in Advances in Optics and Photonics, Vol. 3, Issue 2, pp. 128-160, Mar. 31, 2011), structured light technology using various structured light patterns are described and the corresponding performances are compared. In another example, various design, calibration and implement issues are described in “3-D Computer Vision Using Structured Light: Design, Calibration and Implementation Issues” (DePiero et al., Advances in Computers, Volume 43, Jan. 1, 1996, pages 243-278). Accordingly, the details of the structured light technology are not repeated here.
The structured light technology may be more suitable for 3D imaging of the GI tract than other technologies. However, there are still issues with the intended application for GI tract. For example, most of the structured light applications are intended for stationary object. Therefore, there is no object movement between the captured structured-light image and the regular image. Nevertheless, in the capsule camera application for GI tract imaging, both the capsule camera and the GI parts (e.g. small intestines and colons) may be moving. Therefore, there will be relative movement between the structured-light image and the regular image if they are captured consecutively, in particular under the circumstance of low frame rate. Furthermore, the capsule camera application is a very power-sensitive environment. The use of structured light will consume system power in addition to capturing the regular images. Besides, if one structured light image is taken associated with each regular image, the useful frame rate will be dropped to about half. If the same frame rate of regular images is maintained, the system would have to capture images at twice the regular frame rate and consume roughly twice the power in image capture.
Accordingly, U.S. patent application Ser. No. 14/884,788 discloses technologies for structured light application in the GI tract that can overcome the power issue and capture time difference issue as mentioned above. For example, techniques have been disclosed to allow structured light image captured with shortened frame time and lower power by using an analog-to-digital converter with lower dynamic range. In U.S. patent application Ser. No. 14/884,788, the aspect of controlling the capture of two structured light images for a regular image has been disclosed to further improve the accuracy of depth information associated with a corresponding regular image.
In the application of structured light image, mixed-type images (i.e., structured light image and regular light image) are captured. The structured light image is used to derive the depth information for an associated regular light image. Usually, the depth information does not require having the same spatial resolution requirement or the same bit depth as the corresponding texture information in the regular light image.
There are also other situations of capturing mixed-type images. For example, in U.S. Pat. No. 7,940,973 issued on May 10, 2011, a capsule camera with capture control is disclosed, where the sensor is configured to support the Monitor mode and the Capture mode, where the Monitor mode is operated in low power without the need to store or transmit the generated image. On the other hand, the Capture mode captures images for storage or transmission that will be viewed for diagnosis purposes. Therefore, the images in the Monitor mode can use lower quality and the Capture mode will use higher quality to preserve features of diagnostic interest. In this application, the camera switches to the Capture mode whenever motion or activities are detected. It is desirable to be able to switch from the Monitor mode to the Capture mode as soon as possible when motion or activities are detected. The quick switch may avoid the possibility of missing important diagnostic information during image capture.